688150 


ORC  49-7 
APRIL  1949 


INFINITESIMAL  LOOK-AHEAD  STOPPING  RULES 


SHELDON  M.  ROSS 


OPERATIONS 

RESEARCH 

CENTER 


This  document  has  been  approved 
for  public  rclcoDO  and  solo;  lt» 
distribution  is  unlimited. 


D  n  r 

IP 

m  JUN6  1969 


UUliaLi/cwu  u  ihi 

C 

UNIVERSITY  OF  CALIFORNIA  •  BERKELEY 


jC <-d  b/  the 

CLEARINGHOUS’E 

'or  Federal  Scientific  K  Technical 
Inforrn.jfioi*  Spririgheld  Va  22151 


INFINITESIMAL  LOOK-AHEAD  STOPPING  RULES 


by 


Sheldon  M.  Ross 

Department  of  Industrial  Engineering 
and  Operations  Research 
University  of  California,  Berkeley 


APRIL  1969  ORC  69-7 

This  research  has  been  partially  supported  by  the  Office  of  Naval 
Research  under  Contract  Nonr-222(83) ,  the  National  Science  Foundation 
under  Grant  GP-8695,  and  the  U.  S.  Army  Research  Off ice-Durham  under 
Contract  DA-31-124-ARO-D-331  with  the  University  of  California. 
Reproduction  in  whole  or  in  part  is  permitted  for  any  purpose  of  the 
United  States  Government. 


ABSTRACT 


The  continuous  time  optimal  stopping  problem  Is  considered 
and  an  Infinitesimal  look  ahead  procedure  is  defined. 
Sufficient  conditions  are  then  given  which  ensure  that  this 
procedure,  which  is  the  continuous  time  analogue  of  the  one 
stage  look  ahead  rule  in  the  discrete  time  problem,  is 
optimal.  These  results  are  then  applied  to  a  class  of 
continuous  time  Markov  decision  processes. 


INFINITESIMAL  LOOK-AHEAD  STOPPING  RULES 
by 

S.  M.  Ross 


1.  INTRODUCTION 

Let  X  *  (Xfc  ,  t  >  0)  be  a  Markov  Process  having  stationary  transition 
distributions,  and  sample  paths  which  are  almost  surely  right  continuous  and  have 
only  jump  discontinuities.  The  state  space  S  of  the  process  is  assumed  to  be  a 
Borel  subset  of  a  complete  separable  metric  space  and  we  consider  the  problem  of 
selecting  a  stopping  time  t  maximizing 


(1) 


e‘XTf(X  ) 

T 


-f  e‘Xsc(Xs)ds 
0 


» 


X 

where  f  and  c  are  continuous  real-valued  functions  on  S  ,  \  >  0  ,  and  E 
denotes  expectation  conditional  on  X  »  x  . 

In  the  second  section  of  this  paper  we  show  that  under  certain  conditions  an 
infinitesimal  lock-ahead  procedure  is  optimal.  This  result  generalizes  certain 
discrete  time  results  given  by  Chow-Robbins  [  4  ] .  In  the  third  section  a  related 
approach  is  described  and  the  resultant  procedure  is  shown  to  be  optimal  under 
slightly  more  general  situations.  The  fourth  section  considers  a  class  of 
continuous  time  Markovian  Decision  Processes  for  which  the  criterion  function  is 


-closely  related  to  (1) . 


l 
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2.  INFINITESIMAL  LOOK-AHEAD  STOPPING  RULE 

A  stopping  time  x  is  defined  to  be  any  nonnegative  extended  real-valued 
random  variable  such  that  for  all  t>0,{x<t}  is  contained  in  the  sigma 

field  generated  by  {X  ,0  <  s  <  t}  .  A  stopping  time  x  is  said  to  be  optimal 

-  8  —  -I  ^ 

at  x  e  S  if  EX  e'XT  f(X  *)  -  f  e"Xsc(X  )ds  -  Max  EX[e"XTf(X  )  -/e'XS c(X  )ds 

*  L  T  0  8  J  x  1  T  0  S. 

If  x  is  optimal  at  x  for  every  x  e  S  then  it  is  said  to  be  optimal. 

For  any  Borel  set  B  e  S  ,  let  Pt(B  |  x)  “  P{Xfc  e  B  |  Xq  ■  x}  ,  we  shall 

suppose  at  first  that  X  -  0  .  Let 


(2) 


a(x) 


llm 

h-K)+ 


pcxh)  -  tun 

L  h  . 


x  e  S  . 


We  assume  that  f  and  X  are  such  that  both  the  limit  in  (2)  exists,  and  also 

x  C  3  i 

that  Kolmogorov's  forward  equation  —  E  £a (X  > ]  -  J  f(y)  r-  P  (dy  |  x)  —  holds 

t  **  0 1  t 

(see  Breiman  [3  ]  page  327). 

Lemma  2.1: 

Under  certain  regularity  conditions  (given  below) 


(3) 


t 

EX  /  <s(X  )ds  -  EXf  (X  )  -  f  (x)  V  x  e  S  ,  V  s  >  0  . 

J  8  t 

0 


« 


Proof: 

We  proceed  "formally"  as  follows 
t  t 

(A)  E Xf  a(Xg)ds  - yV[a(Xs)]ds 

0  0 

t 

(5)  .//.<,)  jjj  Pg  (dy  |  x)ds 

0 


! 

I 

l 


i 
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t 

(6) 

"  /f(yj /sTps(dy  1  x)dt 

0 

(7) 

■  J *f(y)  Pfc(dy  |  x)  -  f(x) 

(8) 

-  EXf(Xt)  -  f(x)  . 

Remark: 


From  the  proof  it  is  clear  that  regularity  conditions  are  necessary  for  (i) 
interchanging  the  integral  and  expectation  in  (4),  (ii)  interchanging  the  Integrals 

in  (6),  and  (iii)  justifying  (7).  A  sufficient  condition  for  (i)  is  that 

t 

f  E  |a(X  )|ds  <  00  V  x,t  (see  Doob,  page  63);  Fubini's  theorem  provides  sufficient 
0  8 

conditions  for  (ii) ;  and  a  continuity  condition  of  the  form 
(1  y  -  x 

lim  P  ( (y }  |  x)  *  |  is  sufficient  for  (iii). 

h-K)  (  0  y  y  x 

t 

Let  Wt  -  f(Xt)  -/a(Xg)ds  -  f(XQ)  . 

Lemma  2 . 2 : 


‘[W.  I  X  ,0  <  u  <  s]  *  W  a.s.  V  s  <  t  ,  V 
t  1  u  -  -  s 


x  . 


Proof : 


EX[W  |  X  ,0  <  u  <  s]  -  W  +  E* 
t  1  u’  -  -  s 


f(Xj  -  f(X  )  -  f  a(X  )du  |  X  ,0  <  u  < 

t  s  /  u  u  -  - 


-  W  +  E 

s 


t-s 


-  f(V  -  /  “<Vd“  I x, 


■  W  ,  by  Lemma  3.1  . 


Q.E.D. 


Let  t,.n  ■  min(T,n)  . 


1 
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Lemma  2.3: 

For  all  stopping  times  i  such  that  E3^  -*■  E*W 

rr  °  x*n  t 

T 

c(X  )ds  -  EX  f  (a(X  )  -  c(X  ))ds  +  f(x)  . 

s  J  a  a 

0 

Proof: 

From  Lemmas  2.1  and  2.2  we  have  that  (W^.t  >0)  is  a  zero-mean  Martingale, 
and  it  thus  follows  from  a  Martingale  systems  theorem  (see  Breiman  [2],  p.  302) 

X 

that  E  (W^)  *  0  for  all  bounded  t  .  The  result  then  holds  by  truncation  and 
passage  to  the  limit. 

Q.E.D. 

Remark : 

When  Et  <  "  and  f  bounded  (9)  is  known  as  Dynkin's  Markov  time  identity 
(see  [2],  p.  376). 

We  now  define  the  set  B  e  S  as  follows 


B  =  {x  :  a(x)  -  c(x)  <  0}  . 

X 

We  are  now  ready  for  the  major  theorem.  Let  P  denote  probability  conditional  on 

X  ■  x  . 
o 

Theorem  2.4: 

Suppose  that  E*Wt  q  -*■  E3^  for  all  stopping  times  t  and  all  x  .  If  B 
is  closed  in  the  sense  that  P*{a  t  >  0  :Xt^B}-0  for  all  x  e  B  ,  and  if 

t  ■  inf{t  >0  :  Xfc  e  B} 

Is  finite  with  Prob.  1,  then  it  is  optimal. 

Proof: 

By  Lemma  2.3  we  have  reduced  the  problem  to  one  in  which  there  is  no  reward 
given  for  stopping,  and  there  is  a  cost  a(x)  -  c(x)  per  unit  time  for  being 
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in  state  x  .  The  result  follows  obviously  from  this. 

Q . E ,D . 

* 

What  we  have  done  can  perhaps  best  be  described  as  follows:  We  define  t  , 
the  infinitesimal  look-ahead  (ILA)  rule,  to  be  the  one  which  stops  at  state  x  iff 
the  infinitesimal  look-ahead  gain  is  no  greater  than  the  stopping  gain.  Theorem 

2.4  then  says  that  if  the  set  of  stopping  states  is  closed  (in  the  sense  of  Theorem  1) 

* 

then  x  is  optimal.  This  result  is  clearly  the  continuous  time  analogue  of  the 
Chow-Robbins  result  of  optimality  of  the  one-stage  look-ahead  rule  in  the 
monotonic  case  (see  [4  ]). 

Ex^.uj-e  1: 

Let  ...  be  a  sequence  of  iid  random  variables  with  cdf  F^,  and  let 

(N^ , t  >_  0)  be  a  nonhomogeneous  Poisson  Process,  independent  of  the  Y^'s  ,  and 
with  a  continuous  nonincreasing  rate  function  p(t)  .  Let  ■  max(Y^,  . . . ,  Y^  )  , 
and  consider  the  Markov  Process  {Xfc  «  (t,Mt)  ,  t  >  0}  .  We  take  f(t,m)  ■  m  , 
and  assume  that  c(t,m)  is  nondecreasing  in  both  t  and  m  .  This  is,  of  course, 
the  continuous  time  analogue  of  the  famous  house-selling  problem  (though  for  the 
sake  of  generality  we  have  not  required  that  F(0)  *  0  ,  see  [4  ]  and  [  9  ]). 


a(t,m) 


lim 

h-K) 


/Vh 

1  1 


-  M. 


i 


-  jj(t)E[max(Y  )  -  m] 

1  ,m 

■  u(0  /  ydF(y  +  o)  . 


Since 


J* ydF(y  +  m)  is  nonincreasing  in  m  it  follows  from  Theorem  2.4  that 

y 

x  -  inf{t  >  0  :  p(t) J  ydF(y  +  Mt)  <  c(t,Mt)} 


is  optimal. 

V  |x|dF(x)  <  00  . 
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Example  2 : 


I 


Let  (Nt>t  >0)  be  a  Poisson  Process  with  rate  p  ,  and  consider  the  Markov 
Process  (Xt  -  (t,Nt)  ,  t  >  0)  .  Let  c(t,Nt>  -  N  and  g(t,Nt)  -  -X(T  -  t)2/2  , 
where  T  is  some  fixed  constant.  Then  a(t,Nt)  *  A(T  -  t)  and  from  Theorem  2. A 

fg 

it  follows  that  t  »  inf{t  >  0  :  >  A  (T  -  t) }  is  optimal.  This  problem  arises 

in  determining  the  optimal  intermediate  time  to  dispatch  a  Poisson  Process  (see 
Ross  [10]). 

Generalizing  the  previous  results  to  include  the  case  of  a  discount  factor 
X  >  0  is  easy.  We  consider  a  new  Markov  Process  consisting  of  the  pair 


XJ.  -  (t,Xt)  .  Letting 


(10) 


we  have  that 


(ID 


Thus,  letting 


where 


f’(t,x)  «  e‘Xtf(x) 
c'(t,x)  -  e“Xtc(x) 


o'(t,x)  = 


lim  E 
h-K) 


t.-pQW  -  f(t-x>1 


( 


e'X(t+h)f(xh)  -  e“X 


■  lim  E" 
h-K) 


-At  „  Exle'Xh((V  -  £ 
-  e  lim  E 

h-K) 


tf(x) j 


•t 


“] . 


-  {(t,x)  :  a'(t,x)  -  c(t,x)  <  0} 
■  (x  :  a^(x)  “  c(x)  <  0}  , 


a.  (x)  -  lim  E' 
h-K) 


< 

c 


e"Xhf(Xh)  -  f(x) 


(12) 
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we  have  from  Theorem  2.4  that 


Corollary  2.5; 


If  B  Is  closed,  and  if 


t  -  inf{t  >0  :  e  B^} 


*  + 

is  finite  with  probability  1,  then  t  maximizes  (1)  for  all  x  . 

Example  3; 

Let  (N^,t  >0)  be  a  nonhomogeneous  Poisson  Process  with  rate  y(t)  . 

Suppose  the  reward  for  stopping  when  *  x  is  x  ,  and  the  continuation  rate  at 
■  x  is  c(x)  .  Suppose  further  that  y(t)  is  continuous  nonincreasing,  and 
c(x)  is  continuous  nondecreasing,  and  let  X  be  the  discount  factor. 

The  state  space  is  thus  Xt  ■  (t,Nt>  ,  and 


ax(t,n) 


lim  E 
h+0 


[>%„  -  H  1 
•1 — 


■  lira 
h-K) 


-Xh, 
e  (n 


)h)  -  n 


-nX  +  y(t)  . 


Thus  from  Corollary  2.5  we  have  that 

*  j  -  c(Nt)  J 

t  ■  infjt  >  0  :  > - - j 

is  optimal.  This  example  with  c(x)  =  c  ,  and  y(t)  h  y  was  treated  in  Taylor  [12] 
by  a  different  method. ^ 

^The  regularity  condition  E  V  *  eV  V  x  ,  V  t  must  of  course  be  satisfied, 

T  .n  t 

where  is  appropriately  defined, 

ft  C 

Taylor's  answer  differs  somewhat  from  ours  as  he  supposed  that 
[loge(l  +  X/y)]  ^  -  c/X  was  an  integer. 
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Example  4: 

Consider  Example  1  when  (1) 
and  (ill)  c(t,m)  =  c  .  Then  it 

t*  ■  lnf{t  >  0 

is  optimal. 

Now,  consider  the  same  problem  with  the  exception  that  once  an  offer  is 
rejected  it  is  no  longer  available.  Clearly,  the  optimal  return  for  this  problem 

is  no  greater  than  the  optimal  return  for  the  original  problem.  Thus,  since  the 

* 

optimal  policy  t  is  a  legitimate  policy  for  this  new  problem  (as  it  never  accepts 
an  old  offer)  it  follows  that  it  is  also  optimal  for  this  problem.  This  is  related  to 
certain  results  given  by  Elfvlng  [  7  ]f  and  Siegmund  [13]. 

Comment : 

We  note  here  that  any  general  discount  function  d(t)  may  be  handled  in  the 

-At 


a  discount  factor  A  is  present,  (il)  y(t)  =  y  , 
is  easily  seen  that 


:  y  J * ydF(y  +  m)  -  AM^  <  c) 


same  manner  as  we  dealt  with  e 
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3.  A  RELATED  APPROACH 

e  Xsc(X  )ds  ,  and  let  B  ■  {x  c  S  :  EXZ  <  f(x)  for 
8  C  “ 

all  t  _>  0}  .  Thus,  B  Is  the  set  of  states  at  which  stopping  is  better  than 
continuing  for  any  fixed  amount  of  time. 

Lemma  3.1: 

If  Px{3  t  >_  0  :  i  1}  *0  for  all  x  e  B  ,  then 


Let  Z^  - 


e'Xtf(Xt) 


-1 


L(Zt  |  iu  i  0  <  u  <  s]  <  Z8  a. s.  v  x  e  B  ,  y  s<t 


Proof : 


EX[Zt  |  Xu  ,  0  <  u  <  s]  -  Zfl  +  e‘XsEX 


.“1  (t~ s) 


-X  (u-s) 


e  ’  f(Xt)  -Je  nv“  0/c(Xu)du  -  f(Xg)  |  Xg 


Z  +  e"X®EXs 
s 


t-s 

e  - f(X.  )  -  f  e-Xuc  (X  )du  -  f  (X  )  |  X 

J  U  OB 


-X (t-s) . 


Since  x  e  B  implies  by  hypothesis  that  x  e  B  a.s.,  the  result  follows  from  the 

8 


definition  of  B  . 


Q.E.D. 


Lemma  3.2; 


If 


lim  EXZ 


T.n 


ExZ_r  y  x  ,  v  t  ,  and  if 


Px{3  t>_0:Xt^B}-0  V  xeB, 


then 


EXZ  <  f (x) 

T  — 


V  x  c  B  ,  V  r 
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Proof : 

It  follows  from  Lemma  3.1  that  (Z^  ,  t  ^  0)  Is  a  supermart ingale,  and  so  the 
result  obtains  from  a  standard  supermartingale  systems  theorem  as  in  Lemma  2.3. 

Q.E.D. 

We  have  thus  shown  the  following: 

Theorem  3.3: 

Under  the  conditions  of  Lemma  3.2 

If (x)  x  e  B 

>  f(x)  x  t  B  . 

Proof : 

When  x  e  B  ,  sup  *  f(x)  from  Lemma  3.2.  When  x  t  B  ,  the  result  follows 
from  the  definition  of  B  . 

Q.E.D. 

Let  us  define  the  stopping  time  x  by 

t  -  inf  {t  _>  0  :  Xfc  e  1}  . 

Now,  suppose  that  there  exists  an  optimal  stopping  rule  x  for  x  .  Let 
x^  *  min  (x,x)  ,  then  from  Lemma  3.2  it  follows  that  x^  is  also  optimal  for  x 
But  It  is  easily  seen  that  x^  must  be  a.s.  equal  to  x  .  Thus  if  there  exists  an 
optimal  rule  for  each  x  ,  then  it  follows  that  x  is  optimal.  It  should  also  be 
noted  that  x  is  just  the  continuous  time  analogue  of  the  functional  equation  rule 
(see  Bellman  [1]). 

^All  of  this  is  assuming,  of  course,  the  conditions  of  Lemma  3.2. 
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Some  sufficient  conditions  for  the  existence  of  an  optimal  stopping  rule  are 

given  in  Dynkin  [6]  and  Taylor  (12).  To  determine  the  connection  between  T  and 

*  ,  -  * 

TLA  t  (as  given  by  [ 13 J ) ,  we  first  note  that  B  <  B^  and  so  T  >  T  .  To 

go  the  other  way,  we  need  the  following: 

Corollary  3.4: 

If  the  conditions  of  Corollary  2.5  hold,  then 

* 

T  -  T  . 


Proof : 

We  give  the  proof  for  X  *  0  .  If  x  e  B  ,  then  since  B  is  closed  it 
t 

follows  that  Ex  /  (a(X  )  -  c(X  ))ds  <  0  for  all  t  ,  and  so  the  result  follows 
0  s  3 

from  (9) .  Similar  comments  hold  when  X  >  0  . 

Q.E.D. 

Aside  from  its  own  interest,  the  reason  we  have  considered  this  approach  is 
that  one  may  easily  construct  examples  in  which  B  is  closed  but  B  is  not. 

The  idea  is  also  illuminating,  and  we  paraphrase  it  as  follows:  Call  a  state  bad 
if  stopping  at  that  state  is  better  than  continuing  from  that  state  for  any  fixed 
amount  of  time.  Then  if  this  set  of  states  is  closed  and  if  an  optimal  rule 
exists,  then  the  rule  which  stops  the  first  time  it  enters  a  bad  state  is  optimal. 
Since  a  discrete  time  Markov  Process  may  be  regarded  as  a  continuous  time  Markov 
Process  (with  X^  -  (t,  Xt>)  ,  it  follows  that  this  result  also  holds  for  the 


discrete  time  problem. 
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A.  SOME  RELATED  CRITERIA 


In  this  section  we  consider  the  problem  of  choosing  a  stopping  time  t 
maximizing  either 


EX  e"XTf(X  )  -  f  e”Xsc(X  )ds 
T  0  8 

A  ■  — 1  . . —  \ ,  where  A  >  0 

EX[1  -  rXTl 


E^ff  <XT)  -  /  c(Xa)ds| 


,  where  0  <  E  t  <  0  . 


Criterion  (1A)  represents  expected  total  discounted  return,  and  (15)  the  long  run 
average  cost  per  unit  time,  when  a  sequence  of  independent  stopping  games  are 
played,  each  starting  at  x  .  These  criterions  also  arise  in  connection  with  a 
2-action,  continuous  time  Markovian  Decision  Process  (see  [11])  in  which  the 
"atop"  action  resets  the  process  to  a  fixed  initial  state  x  .  (In  this  connotation 
-f(y)  is  usually  thought  of  as  the  cost  of  resetting  from  state  y) .  For  any 
constant  b  ,  let 


*T(b)  -  (*T  -  b)EX[l  -  e'XT] 


"  T 

:  e"XTf(XT)  - j 
0 


e  Xs(c(X  )  +  Ab)ds  , 

S  ] 


and  let 


ifrT(b)  -  ($t  -  b)E  t 


'  T 

EX  f<Xt)  -/ 
0 


(c(X  )  +  b)ds 
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Lemma  4.1: 

(i)  If  for  sookj  b  ,  0  ■  ip  *(b)  ■  max  ip  (b)  ,  then  b  ■  \p  *  •  max  ip  t  , 

T  T  T  T  T 

and  conversely; 

(ii)  If  for  some  b  ,  0  ■  4  A(b)  ■  max  $  (b)  where  T  ■  (x  :  0  <  E  t  <  *)  , 

t  xer  T 

then  b  ■  $  ^  •  max  4  »  and  conversely. 

t  Ter  T 

Proof : 

Follows  directly  from  (16)  and  (18). 


Remark: 

Part  (i)  of  the  above  Lemma  seems  to  be  new  as  criterion  (14)  does  not  seem 
to  have  been  previously  considered  in  stopping  rule  literature.  Part  (ii)  is  not 
new  and  may  be  found  in  either  Breiman  [  2 ]  or  Taylor  [12], 

We  shall  suppose  for  the  remainder  that  optimal  rules  exist  for  criterions 
(14)  and  (15)  and  we  let  V  *  max  tj>  ,  and  g  *  max  $ 

T  T  xer  T 

Theorem  4.2: 

Under  the  usual  regularity  conditions 

(i)  If  -  (x  :  a^(x)  ~  c(x)  <  VX}  is  closed,  then 
x  -  inf{t  >  0  :  X  e  B^}  is  optimal  for  (14). 

(ii)  If  B2  ■  (x  :  a(x)  -  c(x)  <  g}  is  closed,  then 

A  x  ^ 

x  -  inf{t  >  0  :  Xt  e  B^}  is  optimal  for  (15)  whenever  0  <  E  x  <  09  . 

Proof : 

Follows  directly  from  (17),  (19),  Corollary  2.5  and  Lemma  4.1. 

Q.E.D. 


Remark: 


Since  V  and  g  are  in  general  unknown  the  usefulness  of  Theorem  4.2  is 


mainly  that  it  enables  us  to  determine  the  structure  of  the  optimal  rule. 


Example  5: 

Let  (N^,t  >  0)  be  any  right  continuous  counting  process  with  left  limits. 
Let  c(n)  be  the  cost  rate  when  there  are  n  in  the  system,  and  suppose  that 
c(n)  is  nondecreasing.  Let  f(x)  =  -R  (i.e.,  R  is  the  reset  cost).  Then  the 
related  Markov  Process  is 

Xt  -  (Ng,s  <  t)  , 

and 


ax(Xt)  -  XR  ,  o(Xt)  -  0 

x  -  inf{t  >  0  :  c(Nt)  >  X(R  -  V)}  is  optimal  for  (14),  and 
r  «  inf{t  >  0  j  c(Mt)  >  -g)  is  optimal  for  (15).  Thus,  for  the  average  cost 
case  it  is  optimal  to  reset  the  process  whenever  the  present  cost  rate  is  at 
least  as  large  as  the  optimal  average  cost  per  unit  time. 
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